Object detection via comparison of synchronized pulsed illumination and camera imaging

ABSTRACT

An image processing system may comprise a global shutter camera, an illumination emitter, and a processing system comprising at least one processor and memory. The processing system may be configured to control the image processing system to: control the illumination emitter to illuminate a scene; control the global shutter camera to capture a sequence of images of the scene, wherein the captured sequence of images includes images that are captured without illumination of the scene by the illumination emitter and images that are captured while the scene is illuminated by the illumination emitter; and determine presence of an object in the scene based on comparison of the images captured without illumination of the scene and images captured with illumination of the scene.

FIELD OF THE INVENTION

This invention relates to detecting objects in a scene based oncomparison of images captured by a camera with a synchronized pulsedillumination that may be used in autonomous vehicle applications. Thus,this invention relates generally to detecting objects and, moreparticularly, to systems and methods for detecting objects in a scenefor autonomous vehicle applications.

BACKGROUND

Lidar sensors are being considered for autonomous vehicle applicationwith the potential for object detection and discrimination. Thesefeatures of light detection and ranging (LiDAR) sensors have allowed forthe consideration of autonomous or semi-autonomous vehicle control. Forexample, cruise control systems may incorporate LiDAR for detecting anobject or another vehicle in the pathway of a vehicle. Depending on theapproach speed, the cruise control setting may be automatically adjustedto reduce the speed of the vehicle based on detecting another vehicle inthe pathway of the vehicle. In addition Lidar sensors allow thedetection of small objects at long range (e.g. a brick or block on theroadway at 150 meters distance from the host vehicle) that would pose athreat if it were struck. Other Advanced Driver Assistance Systems(ADAS) sensors (active sensors like radar or passive sensors likecameras) do not have the sensitivity or resolution for this kind ofobject discrimination particularly at long ranges needed for theopportunity and capability to avoid a collision.

Unfortunately, even after years of development by many companies usingdifferent techniques, LiDAR sensors are nowhere near as mature intechnology development as other ADAS sensing technologies (e.g. radarsand cameras) and consequently are on a relative basis, very expensive.Furthermore, uncertainty in the product volumes based upon timelineassumptions for fully autonomous vehicle application add to the costrisk.

SUMMARY

Exemplary embodiments of this disclosure provide systems and methodsthat can be used to improve operation of object detection system. Morespecifically, exemplary embodiments of this disclosure provide an imageprocessing system that includes a global shutter camera and anillumination emitter controlled to provide images in which presenceand/or characteristics of objects can be determined.

According to one exemplary embodiment, an image processing systemincludes: a camera; an illumination emitter; and a processing systemcomprising at least one processor and memory. The processing system isconfigured to control the image processing system to: control theillumination emitter to illuminate a scene; control the camera tocapture a sequence of images of the scene, wherein the captured sequenceof images includes a first image that is captured without illuminationof the scene and a second image that are captured while the scene isilluminated; and determine presence of an object in the scene based oncomparison of illumination of objects first and second images.

In another exemplary embodiment, an image processing system includes: aglobal shutter camera; an illumination emitter; and a processing systemcomprising at least one processor and memory. The processing system isconfigured to control the image processing system to: control theillumination emitter to illuminate a scene; control the global shuttercamera to capture a sequence of images of the scene, wherein thecaptured sequence of images includes images that are captured withoutillumination of the scene by the illumination emitter and images thatare captured while the scene is illuminated by the illumination emitter;and determine presence of an object in the scene based on comparison ofthe images captured without illumination of the scene and imagescaptured with illumination of the scene.

In another exemplary embodiment, the processing system is furtherconfigured to, in response to determining presence of the object in thescene, determine one or more characteristics of the object in the scenebased on comparison of the images captured without illumination of thescene and the images captured with illumination of the scene.

In another exemplary embodiment, comparing of the images capturedwithout illumination of the scene and the images captured withillumination of the scene includes comparing an image captured withoutillumination to a subsequent image captured with illumination.

In another exemplary embodiment, comparing the image captured withoutillumination to the subsequent image captured with illumination includescomparing pixels at the same positions in the image captured withoutillumination and the subsequent image captured with illumination.

In another exemplary embodiment, the processing system is furtherconfigured to: in response to determining presence of the object in thescene, determine a difference between intensity of pixels correspondingto the object in the image captured with illumination and intensity ofpixels corresponding to the object in the image captured withoutillumination; and determine the object as a potential threat for avehicle based on the difference being greater than a set value.

In another exemplary embodiment, the processing system is furtherconfigured to: based on determining that the object is a potentialthreat for the vehicle, determine whether the object is in a path of thevehicle; and based on determining that the object is in the path of thevehicle, control the vehicle to take defensive measured to avoidcollision with the object.

In another exemplary embodiment, the processing system is furtherconfigured to control a vehicle to be positioned within a traveling lanebased on analyzing images of the scene captured without illumination.

In another exemplary embodiment, a field of view of the global shuttercamera exceeds a beam pattern of the illumination emitter.

In another exemplary embodiment, the illumination emitter is a pulsedlaser illumination source.

In another exemplary embodiment, the pulsed laser illumination sourcecomprises a focused diode laser array including a flat and a sphericalsurface, a collimation lens comprising two spherical surfaces, twohorizontal divergence lenses comprising two spherical and twocylindrical surfaces, and two vertical divergence lenses comprising twospherical and two cylindrical surfaces.

In another exemplary embodiment, the image processing system furthercomprises a light detection and ranging (LiDAR) system configured todetect objects within a path of a vehicle.

In another exemplary embodiment, the illumination emitter is controlledto provide illumination for a duration of an integration time of theglobal shutter camera when capturing images while the scene isilluminated by the illumination emitter.

In another exemplary embodiment, a ratio of a number of images capturedwith illumination of the scene to a number of images captured withoutillumination of the scene is less than one half.

In another exemplary embodiment, a computer implemented methodcomprises: controlling an illumination emitter to illuminate a scene;controlling a global shutter camera to capture a sequence of images ofthe scene, wherein the captured sequence of images includes images thatare captured without illumination of the scene by the illuminationemitter and images that are captured while the scene is illuminated bythe illumination emitter; and determining presence of an object in thescene based on comparison of the images captured without illumination ofthe scene and images captured with illumination of the scene.

In another exemplary embodiment, the method further comprises, inresponse to determining presence of the object in the scene, determiningone or more characteristics of the object in the scene based oncomparison of the images captured without illumination of the scene andthe images captured with illumination of the scene.

In another exemplary embodiment, comparing of the images capturedwithout illumination of the scene and the images captured withillumination of the scene includes comparing an image captured withoutillumination to a subsequent image captured with illumination.

In another exemplary embodiment, comparing the image captured withoutillumination to the subsequent image captured with illumination includescomparing pixels at the same positions in the image captured withoutillumination and the subsequent image captured with illumination.

In another exemplary embodiment, the method further comprises: inresponse to determining presence of the object in the scene, determininga difference between intensity of pixels corresponding to the object inthe image captured with illumination and intensity of pixelscorresponding to the object in the image captured without illumination;and determining the object as a potential threat for a vehicle based onthe difference being greater than a set value.

In another exemplary embodiment, the method further comprises: based ondetermining that the object is a potential threat for a vehicle,determining whether the object is in a path of the vehicle; and based ondetermining that the object is in the path of the vehicle, controllingthe vehicle to take defensive measured to avoid collision with theobject.

In another exemplary embodiment, the method further comprisescontrolling a vehicle to be positioned within a traveling lane based onanalyzing images of the scene captured without illumination.

In another exemplary embodiment, a computer-readable non-transitorystorage medium having stored therein a program to be executed by acomputer of an image processing system, the program, when executed,causing the computer to control the image processing system to at least:control an illumination emitter to illuminate a scene; control a globalshutter camera to capture a sequence of images of the scene, wherein thecaptured sequence of images includes images that are captured withoutillumination of the scene by the illumination emitter and images thatare captured while the scene is illuminated by the illumination emitter;and determine presence of an object in the scene based on comparison ofthe images captured without illumination of the scene and imagescaptured with illumination of the scene.

BRIEF DESCRIPTION OF THE DRAWINGS

So that features of the present invention can be understood, a number ofdrawings are described below. It is to be noted, however, that theappended drawings illustrate only particular embodiments of theinvention and are therefore not to be considered limiting of its scope,for the invention may encompass other equally effective embodiments.

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates an exemplary image processing system according to anembodiment of the present disclosure.

FIG. 2A illustrates a simplified circuit diagram of a rolling shutterimager

FIG. 2B illustrates an image sampling of the rolling shutter imager.

FIG. 3A illustrates a simplified circuit diagram of a global shutterimager.

FIG. 3B illustrates an image sampling of the global shutter imager.

FIG. 4 illustrates providing illumination during operation of a rollingshutter imager.

FIG. 5 illustrates providing illumination during operation of a globalshutter imager.

FIG. 6 illustrates characteristics of a laser emitter according to anembodiment of the present disclosure.

FIG. 7A illustrates a perspective view of the lens elements of a lasertransmit lens construction according to an embodiment of the presentdisclosure.

FIG. 7B illustrates a cross sectional view of a laser transmit lensconstruction according to an embodiment of the present disclosure.

FIG. 8 illustrates the laser emitter optic characteristics including thecross section phase in radians along the x and y orientation of aGaussian beam with a wavelength of 0.905 μm.

FIG. 9 illustrates the laser emitter optic characteristics including thepower density irradiance distribution along the x and y orientation of aGaussian beam with a wavelength of 0.905 μm.

FIG. 10 shows a method for detecting presence of an object in a sceneaccording to an embodiment of the present disclosure.

FIG. 11 illustrates an exemplary processing system upon which variousembodiments of the present disclosure(s) may be implemented.

DETAILED DESCRIPTION

Certain example embodiments of this application provide solution(s) thatimprove detection of objects without increasing complexity of thesystem. Embodiments of this disclosure may reduce number of imagingsystem components needed to detect objects of autonomous vehicleapplications.

The problem(s) of single sensor technology does not provide the intravehicle coverage that is (are) solved by adding several sensors andapply situational awareness to meet the customer requirements.

Examples of the present technology optimize the performance ofautomotive ADAS systems. Some examples of the present technologyoptimize the performance of automotive ADAS systems by providing a lowercost approach while providing for long range object detection at highresolution using existing proven technologies that are cost effectivebased upon existing development maturity and market deployment.

Examples of the present technology include a pulsed laser illuminationsource that is directed at the scene. A camera (e.g., a global shuttercamera) is synchronized to the laser pulses such that a comparison ofobject in the scene can be imaged with both active illumination (i.e.laser pulse) and with no illumination (i.e. camera in passive mode withno laser illumination). The comparison of the same imaged pixels atslightly different time intervals yields very high resolution (e.g. 4 MPcamera yields a resolution over the based field of view of less than 2minutes of arc or 0.03° per camera imaged pixel). At this resolution,small objects at large distances can be discriminated objects ofinterests in the images. For example, an 8-inch brick at 150 meters canbe discriminated as an object with 6 pixel array resolution (e.g. 3×2array).

The capability of detecting the objects is based on a comparison of theilluminated light reflecting off of the object to the background light.Since the background light reflection serves as a continuous referencepoint, any object that has a higher reflection based upon the pulsedlight reflection would constitute an object of interest in the vehiclepath. The comparison of the active to passive reflections and the higherthe ratio of illuminated reflection to background reflection, the morelikely that the object has a vertical height component that canconstitute a potential threat. Consequently, once an object is detectedfrom the background noise, standard image enhancement algorithms can beused to determine if the object is in the vehicle path, the estimatedsize and time to collision so as to take defensive measures to avoid acollision.

FIG. 1 illustrates an exemplary system 100 according to an embodiment ofthe present disclosure. The system 100 may perform one or moreoperations related to automotive ADAS systems, such as detecting objectsin a vehicle path. The system 100 may be included in a vehicle capableof performing semi-autonomous and autonomous movement on land, waterand/or in the air.

The system 100 may include a processor 110, memory 120 and one or moresensors 150. While not illustrated in FIG. 1 , the system 100 mayinclude other components such as input device(s), display(s),communication circuitry, etc. The one or more sensors 150 may includeRadio Detection and Ranging (Radar) systems, Light Detection and Ranging(LiDAR) systems and cameras. The system 100 may be configured to receiveand analyze data from the plurality of sensors 150 in order to safelycontrol operation of the vehicle. While not illustrated in FIG. 1 , thesystem 100 may also receive information from external systems (e.g.,other vehicles, remove servers, and/or remote sensors) to be analyzedtogether with the data from the plurality of sensors 150.

The processor 110 comprises hardware (e.g., circuitry) and/or softwareconfigured to perform operations discussed in this application. While asingle processor 110 is illustrates the processor 110 may include aplurality of processors and/or one or more hardware circuits configuredto perform one or more operations discussed in this application. Theprocessor 110 may be coupled to memory configured to store a computerprogram comprising instructions for execution by the processor.

The system 100 may include an illumination emitter 170. The illuminationemitter 170 may be a pulsed laser illumination source. The illuminationemitter 170 may include a plurality of lens elements configured tooutput a beam of desired characteristics. For example, the illuminationemitter 170 may include a focused diode laser array including a flat anda spherical surface, a collimation lens comprising two sphericalsurfaces, two horizontal divergence lenses comprising two spherical andtwo cylindrical surfaces, and two vertical divergence lenses comprisingtwo spherical and two cylindrical surfaces.

In some examples, the illumination emitter 170 may be part of one of thesensors (e.g., a laser part of a light detection and ranging system). Insuch examples, one or more of the sensors 150 not including theillumination emitter (e.g., a camera) may be synchronized with theoperation of the illumination emitter included in another sensor.

The one or more sensors 150 may include cameras including imagersconfigured to capture images. Cameras used in automotive applicationinclude imagers that are of two main types: rolling shutter and globalshutter. The difference in the image sampling is shown by way ofillustration in FIGS. 2A to 3B. FIG. 2A illustrates a simplified circuitdiagram of a rolling shutter imager and FIG. 2B illustrates an imagesampling of the rolling shutter imager. FIG. 3A illustrates a simplifiedcircuit diagram of a global shutter imager and FIG. 3B illustrates animage sampling of the global shutter imager.

In a rolling shutter imager, each column of pixels can have a circuitfor converting photons into a voltage that is read out at the output(see e.g., FIG. 2A). Each row of pixels on the imager begins the nextrows readout after completing the readout of a previous row. With arolling shutter imager, each row of pixels integrates with the same timelength but with a shifted starting time. Thus, each row will start andend its exposure slightly offset in time from its neighbor. An imagecaptured using the rolling shutter imager, can have pixels that arecaptured at different points in time. As shown in FIG. 2B theintegration time for the adjacent rows can overlap but is also offsetfrom each other and the row readout period is offset from adjacent rowswithout overlap.

If an active light source (e.g. laser pulse) is provided, the flash willoverlap with a small portion of the total integration time. FIG. 4 , forexample, shows that a flash provided during a portion of the frameperiod will overlap only certain rows of the imager and/or during onlycertain periods of the integration time. Some pixels of the imager willnot see the light from the flash (see e.g., FIG. 4 ). It may not bepossible to provide a flash during a period of time from a start of theintegration in row 1 to the end of the integration in row n. Inaddition, the flash may overlap integration of rows that below todifferent frames. For example, as shown in FIG. 4 , providing a flashduring integration of row n for a first frame, will also provide theflash during integration of rows 1 and 2 for a second frame.

With a global shutter imager, all pixels are exposed (or pixel charge isintegrated) at the same time. All of the pixels have a simultaneousexposure start and stop times, which provides a real snapshot of a scene(see e.g., FIG. 3B). In global shutter imagers, additional circuitry(see e.g., FIG. 3A) is needed for each pixel to be able to start andstop exposure simultaneously. Since the charge collection (orintegration) function is decoupled from the pixel charge readoutfunction, the integration time and the active light source pulse (e.g.,a laser pulse) can overlap completely in a global shutter imager (seee.g., FIG. 5 ). As shown in FIG. 5 , the duration of the flash maycorrespond to the duration of the integration time. In other examples,the duration of the flash may correspond to only a portion of theintegration time, while still providing the flash to each of the rows.

In certain example embodiments, the light source is synchronized to theglobal shutter camera operation so that the laser emits in a cycle thatpermits consistent field positions to be compared with laser emissionsand without. Examples of the present technology include synchronizingthe light source to the global shutter camera operation so that thelight source is emitted during integration time of each row in theglobal shutter camera. In some examples, a time duration during whichthe light source emits light is approximately the same as theintegration time. In some examples, the time duration of the lightsource being emitted may be set to a predetermined value that is aboveor below the integration time of the camera. In some examples, the timeduration of the light source being emitted may be dynamically adjustedto correspond to the parameters of the camera (e.g., integration time)and/or characteristics of the image captured without the light source.For example, the duration of the light source being emitted may beadjusted based on brightness of the image.

In certain examples, the light source may be a laser emitter. Thecharacteristics of a laser emitter are shown in FIG. 6 according to oneexample of the present disclosure. The emitter optics may provide apredetermined beam pattern (e.g., an elliptical far field beam) having apredetermined horizontal and vertical angles. The emitter optics may bedesigned to create a full width beam pattern (in this case 60°horizontal×10° vertical) using the pulsed laser light frames to compareobject active to passive reflectance at specific pixels within thelarger camera field of view.

FIGS. 7A and 7B illustrates a laser transmit lens construction accordingto one embodiment of the present disclosure. FIG. 7A illustrates aperspective view of the lens elements and FIG. 7B illustrates a crosssectional view of the laser transmit lens construction. The lasertransmit lens construction includes a six lens design providing f/1.75.The laser transmit lens construction may include one Focused Diode LaserArray comprising 1 flat and 1 spherical surface, one Collimationincluding two spherical surfaces, two Horizontal Divergence elements(e.g., 3.9× reducer Galilean telescope) comprising two spherical and twocylindrical surfaces, and two vertical divergence elements (e.g., 1.6×reducer Galilean telescope) comprising two spherical and two cylindricalsurfaces. While not illustrated in FIGS. 7A and 7B one or moreadditional lens elements may be included in the laser transmit lensconstruction to obtain the desired beam pattern. The use of non-asphericlens elements in the laser transmit lens (e.g. cylindrical and sphericalsurfaces) creates the opportunity for a lower lens element constructioncost.

FIGS. 8 and 9 illustrate laser emitter optic characteristics. FIG. 8illustrates the laser emitter optic characteristics including the crosssection phase in radians along the x and y orientation of a Gaussianbeam with a wavelength of 0.905 μm. FIG. 9 illustrates the laser emitteroptic characteristics including the power density irradiancedistribution along the x and y orientation of a Gaussian beam with awavelength of 0.905 μm.

In certain examples, the active laser emitter has a pulse rate of 100nsec and a duty cycle of 0.1%. If the laser is used in sequential flashmode, the emission response rate would peak up to 10K Hz. The globalshutter camera has response times over field of view typically over arange of 30-100 Hz. Therefore, the laser can fire at rates of 100 to 300times faster than the camera field rate and thus illuminate the scenefor many combinations of S/N (i.e. frames of active source light framesto passive frames) for the laser to fire at fractional rates of thecamera so that the scene image can be compared. Examples of therelationships between the laser response, camera response, and number offrames needed to detect an object (e.g., a brick object) at 150 metersare shown in Table 1.

TABLE 1 Integrated Camera frames for Integration brick object LaserCamera Laser (Active to detection at Response Response Frames Passive)150 m 9.9 K HZ 30 Hz 33 of 330 10% 5 9.9 K HZ 30 Hz 100 of 330 30% 5 9.9K HZ 30 Hz 165 of 330 50% 5 9.9 K HZ 60 Hz 33 of 165 20% 3 9.9 K HZ 60Hz 50 of 165 30% 3 9.9 K HZ 60 Hz 83 of 165 50% 3 10.0 K HZ  100 Hz  10of 100 10% 3 10.0 K HZ  100 Hz  25 of 100 25% 2 10.0 K HZ  100 Hz  50 of100 50% 2

The flexibility of the active/passive frames gives many opportunities touse standard image processing to compare the active frames. The activeframes do not have to be sequential as they could be staggered in manycombinations to minimize object movement in field.

For example, a low rate of active to passive frame integration could beinitiated and if a potential object is indicated (by comparison of thepassive to active frame), a higher active to passive frame integrationcould be implemented to confirm object size. At a 100 Hz camera framerate, if the host vehicle was moving at a 100 kph (or 62 mph) velocity,then 2 sequential camera frames would be needed for the equivalent hostvehicle movement of a brick size object at 150 meters distance to bedetected.

The ratio of a number of active frames to a number passive frames may bedynamically adjusted based on: movement parameters of the vehicle (e.g.,speed, velocity) and/or characteristics of objects detected in a sceneby one or more sensors (e.g., distance to object, size of object, typeof object). For example, the number of active frames that are capturedduring a certain period of time may be set to a first value before anobject is detected in a scene and increased to a second value when theobject is detected in a scene. In another example, the number of activeframes that are captured during a certain period of time may be set to afirst value before the vehicle moving at a first speed and may beincreased to a second value when the speed of the vehicle is increased.In another example, the number of active frames that are captured duringa certain period of time may be set to a first value when the vehicle isbeing operated in a semi-autonomous mode and may be set to a secondvalue greater than the first value when the vehicle is operating in afully autonomous mode.

In this manner, a high resolution active laser object detection systemcan be considered with only the additional laser emitter cost over atraditional global shutter camera. In other examples, the highresolution active laser object detection system may be provided togetherwith other sensors of an autonomous vehicle to provide additionalinformation for fusion with other sensor data or for controllingspecific operations of the vehicle.

FIG. 10 shows a method for detecting presence of an object in a scene.In some examples, a processing system including one or more processorsand memory may be configured to control an image processing system toperform one or more operations of the method shown in FIG. 10 .

In operation 510, an illumination emitter is controlled to illuminate ascene. The illumination emitter may be controlled to illuminate thescene only during certain periods of time while a sequence of images iscaptured using a camera, which may be a global shutter camera. Theillumination emitter may be controlled to provide illumination for aduration of an integration time of the global shutter camera.

In operation 520, the camera is controlled to capture images duringillumination of the scene by the illumination emitter and the camera iscontrolled to captured images without illumination of the scene by theillumination emitter in operation 530. The number of images capturedwith illumination and without illumination may be selected based oncharacteristics of the illumination emitter and the camera (e.g., shownin Table 1), detection of an object in the scene, one or morecharacteristics of the object determined by one or more sensors of thesystem, and/or movement parameters of the vehicle (e.g., speed,velocity).

In operation 540, presence of and/or characteristics of an object in thescene are determined based on a comparison of the images captured withillumination to images captured without illumination. The comparison mayinclude comparing illuminated light reflecting from the object to thebackground light from images without illumination. The background lightserves as a continuous reference point. Pixels with reflected light(e.g., difference between pixels in active and passive images) exceedinga threshold may be determined as an object of interest in the vehiclepath. The threshold may be set dynamically based on average pixel valuesin one or more images captured without illumination and/or one or moreimages captured with illumination.

In one example, operation 540 may include first determining presence ofthe object based on the comparison of a first set of images and, if thepresence of the object is determined, determine one or morecharacteristics of the object in the scene based on comparison ofadditional images captured without illumination of the scene andadditional images captured with illumination of the scene. Thecomparison may include comparing an image captured without illuminationto a subsequent image captured with illumination, such that the timedifference between the images being compared is minimized. In someexamples, the comparison may include comparing a first image capturedwithout illumination to (1) a second image captured without illuminationimmediately before the first image and (2) to a third image capturedwithout illumination immediately after the first image.

The comparison may include comparing pixels corresponding to an objectin the subsequent images. The pixels corresponding to the object may bedetected by analyzing only images captured without illumination and/orby comparing images captured with illumination to images capturedwithout illumination. For example, an object may be detected if theilluminated light reflected of the object, as compared to the object inthe image not illuminated by the light source, exceeds a set threshold.

In one example, a plurality of objects within an image may be determinedfrom images captured without illumination and each of the plurality ofobjects (e.g., pixels corresponding to the objects) is analyzed todetermine whether the object is a potential threat by comparing thepixels corresponding to the object in the image captured withoutillumination to pixels corresponding to the object in the image capturedwith illumination. In this example, the initial analysis may determineobjects including objects that are not a threat (e.g., objects paintedon a wall or road) and objects that are a threat (e.g., a brick on theroad), and each of the objects may be further analyzed by using imagescaptured with illumination to determine whether they are a threat.

One or more characteristics of the object (e.g., size, shape, type) mayalso be determined based on the comparison of the images captured withillumination and without illumination. In some examples, the ratio ofactive to passive frames may be increased to improve accuracy of thedetermining the characteristics of the objects.

In operation 550, the vehicle may be controlled based on determiningpresence and/or characteristics of the object. The vehicle may also becontrolled based on data received from other sensors (a LiDAR system,Global Positioning System). In some examples, images captured withoutillumination may be used to extract data for control the vehicle withoutusing images captured with illumination. For example, a vehicle to bepositioned within a traveling lane may be controlled based on analyzingimages of the scene captured without illumination, while objects forcollision avoidance may be detected using both images with and withoutillumination.

In some examples, the system may be configured to, based on determiningthat the object is a potential threat for the vehicle, determine whetherthe object is in a path of the vehicle, and based on determining thatthe object is in the path of the vehicle, control the vehicle to takedefensive measured to avoid collision with the object. For example, thevehicle may be slowed down and/or moved to another lane.

While the examples of the present technology are described withreference to a vehicle, they are not so limited and may be applied toother camera systems installed in other locations. For example, examplesof the present technology may be applicable to cameras installed inaerial vehicles (e.g., drones, planes, autonomous planes), robots,inside or outside of buildings, walls, and traffic lights.

FIG. 11 illustrates an exemplary system 800 upon which embodiments ofthe present disclosure(s) may be implemented. For example, the system800 may perform one or more of the operations described with referenceto FIG. 10 . The system 800 may be included in a vehicle, but is not solimited. The system 800 may be a portable electronic device that iscommonly housed, but is not so limited. The system 800 may include asensor system 900 comprising a camera 910, one or more other sensors 920and a light emitter 930. In some examples, the light emitter 930 maycorrespond to the illumination emitter 170 shown in FIG. 1 . The camera910 may be a global shutter camera. The various components in the system800 may be coupled to each other and/or to a processing system by one ormore communication buses or signal lines 808.

The sensor system 900 may be coupled to a processing system includingone or more processors 812 and memory 814. The processor 812 maycomprise a central processing unit (CPU) or other type of processor.Depending on the configuration and/or type of computer systemenvironment, the memory 814 may comprise volatile memory (e.g., RAM),non-volatile memory (e.g., ROM, flash memory, etc.), or some combinationof the two. Additionally, memory 814 may be removable, non-removable,etc.

In other embodiments, the processing system may comprise additionalstorage (e.g., removable storage 816, non-removable storage 818, etc.).Removable storage 816 and/or non-removable storage 818 may comprisevolatile memory, non-volatile memory, or any combination thereof.Additionally, removable storage 816 and/or non-removable storage 818 maycomprise CD-ROM, digital versatile disks (DVD) or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium which can be used to storeinformation for access by processing system.

As illustrated in FIG. 11 , the processing system may communicate withother systems, components, or devices via peripherals interface 820.Peripherals interface 820 may communicate with an optical sensor 822,external port 824, RC circuitry 826, audio circuity 828 and/or otherdevices. The optical sensor 882 may be a CMOS or CCD image sensor. TheRC circuitry 826 may be coupled to an antenna and allow communicationwith other devices, computers and/or servers using wireless and/or wirednetworks. The system 800 may support a variety of communicationsprotocols, including code division multiple access (CDMA), Global Systemfor Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE),Wi-Fi (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), BLUETOOTH (BLUETOOTH is a registered trademark of BluetoothSig, Inc.), Wi-MAX, a protocol for email, instant messaging, and/or ashort message service (SMS), or any other suitable communicationprotocol, including communication protocols not yet developed as of thefiling date of this document. In an exemplary embodiment, the system 800may be, at least in part, a mobile phone (e.g., a cellular telephone) ora tablet.

A graphics processor 830 may perform graphics/image processingoperations on data stored in a frame buffer 832 or another memory of theprocessing system. Data stored in frame buffer 832 may be accessed,processed, and/or modified by components (e.g., graphics processor 830,processor 812, etc.) of the processing system and/or components of othersystems/devices. Additionally, the data may be accessed (e.g., bygraphics processor 830) and displayed on an output device coupled to theprocessing system. Accordingly, memory 814, removable storage 816,non-removable storage 818, frame buffer 832, or a combination thereof,may comprise instructions that when executed on a processor (e.g., 812,830, etc.) implement a method of processing data (e.g., stored in framebuffer 832) for improved display quality on a display.

The memory 814 may include one or more applications. Examples ofapplications that may be stored in memory 814 include, navigationapplications, telephone applications, email applications, text messagingor instant messaging applications, memo pad applications, address booksor contact lists, calendars, picture taking and management applications,and music playing and management applications. The applications mayinclude a web browser for rendering pages written in the HypertextMarkup Language (HTML), Wireless Markup Language (WML), or otherlanguages suitable for composing webpages or other online content. Theapplications may include a program for browsing files stored in memory.

The memory 814 may include a contact point module (or a set ofinstructions), a closest link module (or a set of instructions), and alink information module (or a set of instructions). The contact pointmodule may determine the centroid or some other reference point in acontact area formed by contact on the touch screen. The closest linkmodule may determine a link that satisfies one or more predefinedcriteria with respect to a point in a contact area as determined by thecontact point module. The link information module may retrieve anddisplay information associated with selected content.

Each of the above identified modules and applications may correspond toa set of instructions for performing one or more functions describedabove. These modules (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules. Thevarious modules and sub-modules may be rearranged and/or combined.Memory 814 may include additional modules and/or sub-modules, or fewermodules and/or sub-modules. Memory 814, therefore, may include a subsetor a superset of the above identified modules and/or sub-modules.Various functions of the system may be implemented in hardware and/or insoftware, including in one or more signal processing and/or applicationspecific integrated circuits.

Memory 814 may store an operating system, such as Darwin, RTXC, LINUX,UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks.The operating system may include procedures (or sets of instructions)for handling basic system services and for performing hardware dependenttasks. Memory 814 may also store communication procedures (or sets ofinstructions) in a communication module. The communication proceduresmay be used for communicating with one or more additional devices, oneor more computers and/or one or more servers. The memory 814 may includea display module (or a set of instructions), a contact/motion module (ora set of instructions) to determine one or more points of contact and/ortheir movement, and a graphics module (or a set of instructions). Thegraphics module may support widgets, that is, modules or applicationswith embedded graphics. The widgets may be implemented using JavaScript,HTML, Adobe Flash, or other suitable computer program languages andtechnologies.

An I/O subsystem 840 may include an imaging sensor controller, a LClight Modulator controller and/or other input/output controller(s). Theother input/output controller(s) may be coupled to other input/controldevices 842, such as one or more buttons. In some alternativeembodiments, input controller(s) may be coupled to any (or none) of thefollowing: a keyboard, infrared port, USB port, and/or a pointer devicesuch as a mouse. The one or more buttons (not shown) may include anup/down button for volume control of the speaker and/or the microphone.The one or more buttons (not shown) may include a push button. The usermay be able to customize a functionality of one or more of the buttons.The touch screen may be used to implement virtual or soft buttons and/orone or more keyboards.

In some embodiments, the system 800 may include circuitry for supportinga location determining capability, such as that provided by the GlobalPositioning System (GPS). The system 800 may include a power system 850for powering the various components. The power system 850 may include apower management system, one or more power sources (e.g., battery,alternating current (AC)), a recharging system, a power failuredetection circuit, a power converter or inverter, a power statusindicator (e.g., a light-emitting diode (LED)) and any other componentsassociated with the generation, management and distribution of power inportable devices. The system 800 may also include one or more externalports 824 for connecting the system 800 to other devices.

Portions of the present invention may be comprised of computer-readableand computer-executable instructions that reside, for example, in aprocessing system and which may be used as a part of a general purposecomputer network (not shown). It is appreciated that processing systemis merely exemplary. As such, the embodiment in this application canoperate within a number of different systems including, but not limitedto, general-purpose computer systems, embedded computer systems, laptopcomputer systems, hand-held computer systems, portable computer systems,stand-alone computer systems, game consoles, gaming systems or machines(e.g., found in a casino or other gaming establishment), or onlinegaming systems.

Embodiments of the subject matter and the functional operationsdescribed herein can be implemented in one or more of the following:digital electronic circuitry; tangibly-embodied computer software orfirmware; computer hardware, including the structures disclosed in thisspecification and their structural equivalents; and combinationsthereof. Such embodiments can be implemented as one or more modules ofcomputer program instructions encoded on a tangible non-transitorystorage medium for execution by, or to control the operation of, dataprocessing apparatus (i.e., one or more computer programs). The computerstorage medium can be one or more of: a machine-readable storage device,a machine-readable storage substrate, a random or serial access memorydevice, and combinations thereof.

The exemplary embodiments of the present disclosure provide theinvention(s), including the best mode, and also to enable a personskilled in the art to practice the invention, including making and usingany devices or systems and performing any incorporated methods. Whilespecific exemplary embodiments of the present invention(s) are disclosedherein, it should be understood that modifications, substitutions andalternatives may be apparent to one of ordinary skill in the art and canbe made without departing from the scope of this application.

1. An image processing system comprising: a global shutter camera; anillumination emitter; and a processing system comprising at least oneprocessor and memory, the processing system configured to control theimage processing system to: control the illumination emitter toilluminate a scene; control the global shutter camera to capture asequence of images of the scene, wherein the captured sequence of imagesincludes images that are captured without illumination of the scene bythe illumination emitter and images that are captured while the scene isilluminated by the illumination emitter; and determine presence of anobject in the scene based on comparison of the images captured withoutillumination of the scene and images captured with illumination of thescene.
 2. The image processing system of claim 1, wherein the processingsystem is further configured to, in response to determining presence ofthe object in the scene, determine one or more characteristics of theobject in the scene based on comparison of the images captured withoutillumination of the scene and the images captured with illumination ofthe scene.
 3. The image processing system of claim 1, wherein comparingof the images captured without illumination of the scene and the imagescaptured with illumination of the scene includes comparing an imagecaptured without illumination to a subsequent image captured withillumination.
 4. The image processing system of claim 3, whereincomparing the image captured without illumination to the subsequentimage captured with illumination includes comparing pixels at the samepositions in the image captured without illumination and the subsequentimage captured with illumination.
 5. The image processing system ofclaim 1, wherein the processing system is further configured to: inresponse to determining presence of the object in the scene, determine adifference between intensity of pixels corresponding to the object inthe image captured with illumination and intensity of pixelscorresponding to the object in the image captured without illumination;and determine the object as a potential threat for a vehicle based onthe difference being greater than a set value.
 6. The image processingsystem of claim 5, wherein the processing system is further configuredto: based on determining that the object is a potential threat for thevehicle, determine whether the object is in a path of the vehicle; andbased on determining that the object is in the path of the vehicle,control the vehicle to take defensive measured to avoid collision withthe object.
 7. The image processing system of claim 6, wherein theprocessing system is further configured to control a vehicle to bepositioned within a traveling lane based on analyzing images of thescene captured without illumination.
 8. The image processing system ofclaim 1, wherein a field of view of the global shutter camera exceeds abeam pattern of the illumination emitter.
 9. The image processing systemof claim 1, wherein the illumination emitter is a pulsed laserillumination source.
 10. The image processing system of claim 9, whereinthe pulsed laser illumination source comprises a focused diode laserarray including a flat and a spherical surface, a collimation lenscomprising two spherical surfaces, two horizontal divergence lensescomprising two spherical and two cylindrical surfaces, and two verticaldivergence lenses comprising two spherical and two cylindrical surfaces.11. The image processing system of claim 1, further comprising a lightdetection and ranging (LiDAR) system configured to detect objects withina path of a vehicle.
 12. The image processing system of claim 1, whereinthe illumination emitter is controlled to provide illumination for aduration of an integration time of the global shutter camera whencapturing images while the scene is illuminated by the illuminationemitter.
 13. The image processing system of claim 1, wherein a ratio ofa number of images captured with illumination of the scene to a numberof images captured without illumination of the scene is less than onehalf.
 14. A computer implemented method comprising: controlling anillumination emitter to illuminate a scene; controlling a global shuttercamera to capture a sequence of images of the scene, wherein thecaptured sequence of images includes images that are captured withoutillumination of the scene by the illumination emitter and images thatare captured while the scene is illuminated by the illumination emitter;and determining presence of an object in the scene based on comparisonof the images captured without illumination of the scene and imagescaptured with illumination of the scene.
 15. The method of claim 14,further comprising, in response to determining presence of the object inthe scene, determining one or more characteristics of the object in thescene based on comparison of the images captured without illumination ofthe scene and the images captured with illumination of the scene. 16.The method of claim 14, wherein comparing of the images captured withoutillumination of the scene and the images captured with illumination ofthe scene includes comparing an image captured without illumination to asubsequent image captured with illumination.
 17. The method of claim 16,wherein comparing the image captured without illumination to thesubsequent image captured with illumination includes comparing pixels atthe same positions in the image captured without illumination and thesubsequent image captured with illumination.
 18. The method of claim 14,further comprising: in response to determining presence of the object inthe scene, determining a difference between intensity of pixelscorresponding to the object in the image captured with illumination andintensity of pixels corresponding to the object in the image capturedwithout illumination; and determining the object as a potential threatfor a vehicle based on the difference being greater than a set value.19. The method of claim 18, further comprising: based on determiningthat the object is a potential threat for a vehicle, determining whetherthe object is in a path of the vehicle; and based on determining thatthe object is in the path of the vehicle, controlling the vehicle totake defensive measured to avoid collision with the object.
 20. Acomputer-readable non-transitory storage medium having stored therein aprogram to be executed by a computer of an image processing system, theprogram, when executed, causing the computer to control the imageprocessing system to at least: control an illumination emitter toilluminate a scene; control a global shutter camera to capture asequence of images of the scene, wherein the captured sequence of imagesincludes images that are captured without illumination of the scene bythe illumination emitter and images that are captured while the scene isilluminated by the illumination emitter; and determine presence of anobject in the scene based on comparison of the images captured withoutillumination of the scene and images captured with illumination of thescene.